The present invention relates generally to the field of processing images, and more particularly, to the field of remote conferencing applications.
This application incorporates by reference the xe2x80x9cInteractive Projected Video Image Display Systemxe2x80x9d disclosed under U.S. Pat. No. 5,528,263 (Platzker et al.) as if set forth at length herein.
Definitions
Background Image A computer-generated template, often from another active application (such as Microsoft(copyright) PowerPoint(copyright)1), incorporated into the Computer Display Image.
1 Microsoft and PowerPoint are registered trademarks of Microsoft Corporation. 
A Committed Image A composite of the Background Image and all Writing Images to be saved into some suitable medium, such as storage component, transmission channel, or to a compatible software application.
Computer Display Image The display presented by a computer for human viewing on a device such as a monitor or Projection Surface employing technologies such as VGA, SVGA, XGA and others.
Image Sensor An optical device such as a video camera, digital camera or other imaging technology capable of viewing a Writing Surface with Local Markings.
Local Markings Markings (xe2x80x9cLocal Writingsxe2x80x9d and/or xe2x80x9cLocal Erasuresxe2x80x9d) made on the Writing Surface of the site of reference.
Local Updates Stream of data packets containing compressed representations of changes made to Local Markings.
Markings Writings and annotations (collectively, xe2x80x9cWritingsxe2x80x9d) and/or deletions (xe2x80x9cErasuresxe2x80x9d) made by a human presenter on a Writing Surface.
Projection Surface A surface upon which the Computer Display Image created by a computer-controlled projection may appear; in the present invention, this will substantially overlap the Writing Surface.
Projections Visual information that appears in the Viewed Image as a result of projecting the Computer Display Image onto the Writing Surface (distinguished from Markings and Viewed Image Interference); in the context of the present invention Projections are relevant insomuch as they appear similar to physical Markings or otherwise complicate the task of detecting and processing Markings.
Remote Markings Markings (xe2x80x9cRemote Writingsxe2x80x9d and/or xe2x80x9cRemote Erasuresxe2x80x9d) made on the Writing Surfaces of non-local sites.
Remote Updates Stream of data packets containing compressed representations of changes made to Remote Markings.
Stored Viewed Image The most recent Viewed Image modified to exclude Viewed Image Interference and retained for purposes of detecting changes to Local Markings in a comparison with a Viewed Image.
Stored Writing Image The most recent Writing Image retained for purposes of encoding Local Updates in a comparison with an updated Writing Image.
Viewed Image The image acquired (or xe2x80x9cseenxe2x80x9d) by the Image Sensor and made available as digital information to computational resources (software/hardware).
Viewed Image Interference Physical objects that are interposed between the Image Sensor and the Writing Surface and therefore appear in the Viewed Image, for example, a writer""s arm or body (distinguished from Markings and Projections).
Warping A transformation performed on an image based on a mapping between two geometrical coordinate spaces; in the present invention Viewed Images (or portions thereof) are transformed in this manner to a predefined display coordinate space and projected images (or portions thereof) are transformed to the coordinate space of the Viewed Images (using both Warping and optional xe2x80x9cscalingxe2x80x9d to overcome differences in pixel-resolution); the geometric mapping is obtained through a process of calibration.
Writing Images Internal representations of the Local Markings or Remote Markings at any point in time, one Writing Image per participating transmitting site.
Writing Surface Any surface suitable for human writing, such as a whiteboard, paper flip-chart or sheets of paper; in the present invention, this will also be the Projection Surface, if employed.
Business, academic and other professional meetings are held to impart information to, and solicit ideas from, the attendees. The convener of a meeting usually seeks participation from the attendees, and it is an effective use of expensive meeting time to capture and record the key thoughts and ideas of the presenters and participants for future reference. It is particularly effective to record and display key points, numbers, etc., dynamically as these are forthcoming from participants at a meeting or seminar. An issue is how to record and display this information most effectively with minimal distraction. The transcription by hand of information written on a vertical board is known to every school child.
Such data are ephemeral, however, and must eventually be erased. Traditionally, each participant takes his or her own notes, each copying essentially the same material. Paradoxically, valuable information may be missed by one or several or many of participants because of the diversion of their attention to note-taking, and there may be errors in the transcription.
One approach to solving this problem is disclosed in the U.S. Pat. No. 5,528,263 previously incorporated hereinabove by reference, the xe2x80x9cInteractive Projected Video Image Display System.xe2x80x9d In this patent beginning at column 6, line 60 a function is described to enable a computer to capture updated images. This system has been commercialized as Digital Flipchart(trademark) (xe2x80x9cDFCxe2x80x9d). The system disclosed in the Platzker et al. patent and the one commercialized by Tegrity can be used at only one location and cannot capture an updated image without blanking the screen and cannot ignore transient objects in the field of view.
Video technology could solve the problem and limitations of the prior art, but would increase the cost and complexity of linking remote sites. Linking remote sites would allow remote participants to view written information from other sites in real-time. Using video technology to this end, the presenter must be careful not to block the view unduly, adding an unnatural constraint to a presentation. On the remote end, video images of a person""s arm, blouse or tie are unnatural, unsatisfactory, uninformative, distracting at best and, at worst, an annoyance that detracts from the value of the presentation to remote viewers. Moreover, such useless video data further burdens the transmission channel. Combining video images from dispersed locations adds even more complexity.
Transmitting even compressed video uses substantial bandwidth. Transmitting compressed video of images containing motion uses even more bandwidth.
The present invention is an apparatus and method for providing a composite image on a standard whiteboard, flipchart or other Writing Surface""s location at a first site, consisting of Local Markings and a projection of only the meaningful changes (i.e., Writings and Erasures) made to at least a second Writing Surface located at a second site. The result is an appearance on the local Projection Surface of a combination of physical Local Markings together with a projection of the remote sites"" respective Markings. Thus, the same composite image appears on the Projection Surface at every site, although local viewers see the actual physical Markings made locally, as opposed to the projected Remote Markings. To human perception, everyone at every connected site is seeing the xe2x80x9csamexe2x80x9d image, differing only in size and to the degree imposed by the technical capabilities of the local projector. The composite image also incorporates the Background Image if one is generated within the computer.
In general, each site may utilize an Image Sensor connected to a standard PC-type processor. The process, when employed in such a site, continuously xe2x80x9clooksxe2x80x9d (i.e., grabs Viewed Images via the Image Sensor) at the present composition of the Writing Surface. It also continuously monitors the present composition of the Computer Display Image, if it is projected on the Writing Surface. These image are analyzed to detect the existence and precise locations of three kinds of information:
1. Viewed Image Interferencexe2x80x94these areas are detected, ignored and therefore not encoded or transmitted and no bandwidth is consumed by them.
2. Projected xe2x80x9cwritingsxe2x80x9dxe2x80x94 information that represents projected xe2x80x9cobjectsxe2x80x9d sensed by the Image Sensor (e.g., Writings from other stations which are projected by a projector). These Projections are similarly detected, ignored and withheld from transmission.
3. Local changesxe2x80x94changes to Local Markings which appeared after the previous analysis. These local changes are processed. Remote sites (if any) may utilize a different display resolution (the number of pixels in the VGA). Local changes undergo a process of geometric adaptation (Warping), which transforms them to a common display resolution. The local changes are then encoded in a compressed format as Local Updates and sent to other, remote sites for display (with optional buffering of the transmitted data), and/or appended to a stored data stream. The result of the process is that each site displays the Writings of all other parties. This display may be projected on the same physical Projection Surface upon which Local Markings are written. Consequently, all of the displays at each interconnected site contain all of the information recorded elsewhere. Local Writings (i.e., locally recorded notations) are not displayed (or projected). Users have a xe2x80x9ccommit capturexe2x80x9d option whereby the resulting composite Committed Image of the Writings from all sites may be saved. There may also be xe2x80x9creceiverxe2x80x9d only stations, i.e., sites which do not transmit Local Updates, if any, and need not be equipped with an Image Sensor or a Writing Surface. These sites merge and display Markings received from other physical sites. Optionally, such sites can be made to operate using standard, widely available Internet browsing software without the need for additional, special-purpose hardware or software components.
In an optional embodiment, a remote site can be configured as display-only. The composite image for display or projection at such a display-only (or receive-only) site would consist of the assembled, ongoing changes to Remote Markings and Background Image without any physical Local Markings comprising part of the image.
The remote conferencing capabilities of the invention are facilitated by the innovation of a method of using a computer to isolate changes to Markings, filtering out transient data, and compressing the result for transmission and/or storage. The computer processes the Viewed Image signals, or xe2x80x9cframes,xe2x80x9d representing the images appearing in the viewing field of the local Image Sensor indicative of Markings made on the local Writing Surface. The meaningful portion of this continuous sequence of frames are those not indicative of the presence of Viewed Image Interference, should any appear transiently in the local viewing field, or of Projections. The method detects changes between successive frames. These frames are filtered to reject those signals below a predetermined spatial frequency representing obstructions, and the successive, filtered frames are further examined to detect changes indicative of the Writings or Erasures made on the local Writing Surface as distinguished from transient changes characteristic of Viewed Image Interference. These changes are further examined to eliminate those that are caused by changes to the Projections, if the Computer Display Image is projected. The result is a compressed representation of changes to Markings, suitable for decompression and display at the receiving end. As such, the method may be termed a xe2x80x9cwriting codec.xe2x80x9d
Representations of the successive changes may be stored for later playback, locally or remotely. In one embodiment, an image may be transmitted from time to time even without changes to permit a late joining site or one which is restarting to develop a composite image.